Effective sample size: A measure of individual uncertainty in predictions.

Clinical prediction models are estimated using a sample of limited size from the target population, leading to uncertainty in predictions, even when the model is correctly specified. Generally, not all patient profiles are observed uniformly in model development. As a result, sampling uncertainty varies between individual patients' predictions. We aimed to develop an intuitive measure of individual prediction uncertainty. The variance of a patient's prediction can be equated to the variance of the sample mean outcome in n ∗ $$ {n}_{\ast } $$ hypothetical patients with the same predictor values. This hypothetical sample size n ∗ $$ {n}_{\ast } $$ can be interpreted as the number of similar patients n eff $$ {n}_{\mathrm{eff}} $$ that the prediction is effectively based on, given that the model is correct. For generalized linear models, we derived analytical expressions for the effective sample size. In addition, we illustrated the concept in patients with acute myocardial infarction. In model development, n eff $$ {n}_{\mathrm{eff}} $$ can be used to balance accuracy versus uncertainty of predictions. In a validation sample, the distribution of n eff $$ {n}_{\mathrm{eff}} $$ indicates which patients were more and less represented in the development data, and whether predictions might be too uncertain for some to be practically meaningful. In a clinical setting, the effective sample size may facilitate communication of uncertainty about predictions. We propose the effective sample size as a clinically interpretable measure of uncertainty in individual predictions. Its implications should be explored further for the development, validation and clinical implementation of prediction models.

Landmarking 2.0: Bridging the gap between joint models and landmarking.

The problem of dynamic prediction with time-dependent covariates, given by biomarkers, repeatedly measured over time, has received much attention over the last decades. Two contrasting approaches have become in widespread use. The first is joint modeling, which attempts to jointly model the longitudinal markers and the event time. The second is landmarking, a more pragmatic approach that avoids modeling the marker process. Landmarking has been shown to be less efficient than correctly specified joint models in simulation studies, when data are generated from the joint model. When the mean model is misspecified, however, simulation has shown that joint models may be inferior to landmarking. The objective of this article is to develop methods that improve the predictive accuracy of landmarking, while retaining its relative simplicity and robustness. We start by fitting a working longitudinal model for the biomarker, including a temporal correlation structure. Based on that model, we derive a predictable time-dependent process representing the expected value of the biomarker after the landmark time, and we fit a time-dependent Cox model based on the predictable time-dependent covariate. Dynamic predictions based on this approach for new patients can be obtained by first deriving the expected values of the biomarker, given the measured values before the landmark time point, and then calculating the predicted probabilities based on the time-dependent Cox model. We illustrate the approach in predicting overall survival in liver cirrhosis patients based on prothrombin index.

Analysis of time-to-event for observational studies: Guidance to the use of intensity models.

This paper provides guidance for researchers with some mathematical background on the conduct of time-to-event analysis in observational studies based on intensity (hazard) models. Discussions of basic concepts like time axis, event definition and censoring are given. Hazard models are introduced, with special emphasis on the Cox proportional hazards regression model. We provide check lists that may be useful both when fitting the model and assessing its goodness of fit and when interpreting the results. Special attention is paid to how to avoid problems with immortal time bias by introducing time-dependent covariates. We discuss prediction based on hazard models and difficulties when attempting to draw proper causal conclusions from such models. Finally, we present a series of examples where the methods and check lists are exemplified. Computational details and implementation using the freely available R software are documented in Supplementary Material. The paper was prepared as part of the STRATOS initiative.

Prediction meets causal inference: the role of treatment in clinical prediction models.

In this paper we study approaches for dealing with treatment when developing a clinical prediction model. Analogous to the estimand framework recently proposed by the European Medicines Agency for clinical trials, we propose a 'predictimand' framework of different questions that may be of interest when predicting risk in relation to treatment started after baseline. We provide a formal definition of the estimands matching these questions, give examples of settings in which each is useful and discuss appropriate estimators including their assumptions. We illustrate the impact of the predictimand choice in a dataset of patients with end-stage kidney disease. We argue that clearly defining the estimand is equally important in prediction research as in causal inference.

On the relation between the cause-specific hazard and the subdistribution rate for competing risks data: The Fine-Gray model revisited.

The Fine-Gray proportional subdistribution hazards model has been puzzling many people since its introduction. The main reason for the uneasy feeling is that the approach considers individuals still at risk for an event of cause 1 after they fell victim to the competing risk of cause 2. The subdistribution hazard and the extended risk sets, where subjects who failed of the competing risk remain in the risk set, are generally perceived as unnatural . One could say it is somewhat of a riddle why the Fine-Gray approach yields valid inference. To take away these uneasy feelings, we explore the link between the Fine-Gray and cause-specific approaches in more detail. We introduce the reduction factor as representing the proportion of subjects in the Fine-Gray risk set that has not yet experienced a competing event. In the presence of covariates, the dependence of the reduction factor on a covariate gives information on how the effect of the covariate on the cause-specific hazard and the subdistribution hazard relate. We discuss estimation and modeling of the reduction factor, and show how they can be used in various ways to estimate cumulative incidences, given the covariates. Methods are illustrated on data of the European Society for Blood and Marrow Transplantation.

Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration.

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta-analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6-month mortality based on individual patient data using meta-analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.

MAGE-A3 immunotherapeutic as adjuvant therapy for patients with resected, MAGE-A3-positive, stage III melanoma (DERMA): a double-blind, randomised, placebo-controlled, phase 3 trial.

BACKGROUND: Despite newly approved treatments, metastatic melanoma remains a life-threatening condition. We aimed to evaluate the efficacy of the MAGE-A3 immunotherapeutic in patients with stage IIIB or IIIC melanoma in the adjuvant setting. METHODS: DERMA was a phase 3, double-blind, randomised, placebo-controlled trial done in 31 countries and 263 centres. Eligible patients were 18 years or older and had histologically proven, completely resected, stage IIIB or IIIC, MAGE-A3-positive cutaneous melanoma with macroscopic lymph node involvement and an Eastern Cooperative Oncology Group performance score of 0 or 1. Randomisation and treatment allocation at the investigator sites were done centrally via the internet. We randomly assigned patients (2:1) to receive up to 13 intramuscular injections of recombinant MAGE-A3 with AS15 immunostimulant (MAGE-A3 immunotherapeutic; 300 µg MAGE-A3 antigen plus 420 µg CpG 7909 reconstituted in AS01B to a total volume of 0·5 mL), or placebo, over a 27-month period: five doses at 3-weekly intervals, followed by eight doses at 12-weekly intervals. The co-primary outcomes were disease-free survival in the overall population and in patients with a potentially predictive gene signature (GS-positive) identified previously and validated here via an adaptive signature design. The final analyses included all patients who had received at least one dose of study treatment; analyses for efficacy were in the as-randomised population and for safety were in the as-treated population. This trial is registered with ClinicalTrials.gov, number NCT00796445. FINDINGS: Between Dec 1, 2008, and Sept 19, 2011, 3914 patients were screened, 1391 randomly assigned, and 1345 started treatment (n=895 for MAGE-A3 and n=450 for placebo). At final analysis (data cutoff May 23, 2013), median follow-up was 28·0 months [IQR 23·3-35·5] in the MAGE-A3 group and 28·1 months [23·7-36·9] in the placebo group. Median disease-free survival was 11·0 months (95% CI 10·0-11·9) in the MAGE-A3 group and 11·2 months (8·6-14·1) in the placebo group (hazard ratio [HR] 1·01, 0·88-1·17, p=0·86). In the GS-positive population, median disease-free survival was 9·9 months (95% CI 5·7-17·6) in the MAGE-A3 group and 11·6 months (5·6-22·3) in the placebo group (HR 1·11, 0·83-1·49, p=0·48). Within the first 31 days of treatment, adverse events of grade 3 or worse were reported by 126 (14%) of 894 patients in the MAGE-A3 group and 56 (12%) of 450 patients in the placebo group, treatment-related adverse events of grade 3 or worse by 36 (4%) patients given MAGE-A3 vs six (1%) patients given placebo, and at least one serious adverse event by 14% of patients in both groups (129 patients given MAGE-A3 and 64 patients given placebo). The most common adverse events of grade 3 or worse were neoplasms (33 [4%] patients in the MAGE-A3 group vs 17 [4%] patients in the placebo group), general disorders and administration site conditions (25 [3%] for MAGE-A3 vs four [<1%] for placebo) and infections and infestations (17 [2%] for MAGE-A3 vs seven [2%] for placebo). No deaths were related to treatment. INTERPRETATION: An antigen-specific immunotherapeutic alone was not efficacious in this clinical setting. Based on these findings, development of the MAGE-A3 immunotherapeutic for use in melanoma has been stopped. FUNDING: GlaxoSmithKline Biologicals SA.

Understanding Landmarking and Its Relation with Time-Dependent Cox Regression.

Time-dependent Cox regression and landmarking are the two most commonly used approaches for the analysis of time-dependent covariates in time-to-event data. The estimated effect of the time-dependent covariate in a landmarking analysis is based on the value of the time-dependent covariate at the landmark time point, after which the time-dependent covariate may change value. In this note we derive expressions for the (time-varying) regression coefficient of the time-dependent covariate in the landmark analysis, in terms of the regression coefficient and baseline hazard of the time-dependent Cox regression. These relations are illustrated using simulation studies and using the Stanford heart transplant data.

Testing interaction between treatment and high-dimensional covariates in randomized clinical trials.

In this paper, we considered different methods to test the interaction between treatment and a potentially large number (p) of covariates in randomized clinical trials. The simplest approach was to fit univariate (marginal) models and to combine the univariate statistics or p-values (e.g., minimum p-value). Another possibility was to reduce the dimension of the covariates using the principal components (PCs) and to test the interaction between treatment and PCs. Finally, we considered the Goeman global test applied to the high-dimensional interaction matrix, adjusted for the main (treatment and covariates) effects. These tests can be used for personalized medicine to test if a large set of biomarkers can be useful to identify a subset of patients who may be more responsive to treatment. We evaluated the performance of these methods on simulated data and we applied them on data from two early phases oncology clinical trials.

Dynamic frailty models based on compound birth-death processes.

Frailty models are used in survival analysis to model unobserved heterogeneity. They accommodate such heterogeneity by the inclusion of a random term, the frailty, which is assumed to multiply the hazard of a subject (individual frailty) or the hazards of all subjects in a cluster (shared frailty). Typically, the frailty term is assumed to be constant over time. This is a restrictive assumption and extensions to allow for time-varying or dynamic frailties are of interest. In this paper, we extend the auto-correlated frailty models of Henderson and Shimakura and of Fiocco, Putter and van Houwelingen, developed for longitudinal count data and discrete survival data, to continuous survival data. We present a rigorous construction of the frailty processes in continuous time based on compound birth-death processes. When the frailty processes are used as mixtures in models for survival data, we derive the marginal hazards and survival functions and the marginal bivariate survival functions and cross-ratio function. We derive distributional properties of the processes, conditional on observed data, and show how to obtain the maximum likelihood estimators of the parameters of the model using a (stochastic) expectation-maximization algorithm. The methods are applied to a publicly available data set.

Comparison of stopped Cox regression with direct methods such as pseudo-values and binomial regression.

By far the most popular model to obtain survival predictions for individual patients is the Cox model. The Cox model does not make any assumptions on the underlying hazard, but it relies heavily on the proportional hazards assumption. The most common ways to circumvent this robustness problem are 1) to categorize patients based on their prognostic risk score and to base predictions on Kaplan-Meier curves for the risk categories, or 2) to include interactions with the covariates and suitable functions of time. Robust estimators of the t(0)-year survival probabilities can also be obtained from a "stopped Cox" regression model, in which all observations are administratively censored at t(0). Other recent approaches to solve this robustness problem, originally proposed in the context of competing risks, are pseudo-values and direct binomial regression, based on unbiased estimating equations. In this paper stopped Cox regression is compared with these direct approaches. This is done by means of a simulation study to assess the biases of the different approaches and an analysis of breast cancer data to get some feeling for the performance in practice. The tentative conclusion is that stopped Cox and direct models agree well if the follow-up is not too long. There are larger differences for long-term follow-up data. There stopped Cox might be more efficient, but less robust.

Frailties in multi-state models: Are they identifiable? Do we need them?

The inclusion of latent frailties in survival models can serve two purposes: (1) the modelling of dependence in clustered data, (2) explaining lack of fit of univariate survival models, like deviation from the proportional hazards assumption. Multi-state models are somewhere between univariate data and clustered data. Frailty models can help in understanding the dependence in sequential transitions (like in clustered data) and can be useful in explaining some strange phenomena in the effect of covariates in competing risks models (like in univariate data). The (im)possibilities of frailty models will be exemplified on a data set of breast cancer patients with death as absorbing state and local recurrence and distant metastasis as intermediate events.

The role of empirical Bayes methodology as a leading principle in modern medical statistics.

This paper reviews and discusses the role of Empirical Bayes methodology in medical statistics in the last 50 years. It gives some background on the origin of the empirical Bayes approach and its link with the famous Stein estimator. The paper describes the application in four important areas in medical statistics: disease mapping, health care monitoring, meta-analysis, and multiple testing. It ends with a warning that the application of the outcome of an empirical Bayes analysis to the individual "subjects" is a delicate matter that should be handled with prudence and care.

From model building to validation and back: a plea for robustness.

This paper is the written version of the President's invited lecture speaker at the International Society for Clinical Biostatistics conference in Munich in 2013. The paper takes the stand of clinician and patient who are in need of a reliable prognostic model for the planning of treatment and patient care during the follow-up after the initial treatment. The paper discusses (i) the need for grouping of data; (ii) the lack of robustness of the Cox model; (iii) the robust approach to repeated measures; and (iv) the robust handling of time-dependent covariates (biomarkers) in dynamic survival analysis.

Validation, calibration and refitting of a familial breast cancer model in sisterships: a case study in the Swedish sisters data.

In Sweden, a unique data set has been compiled with breast cancer incidence in all sisterships with at least two sisters born between 1932 and 2001, and the effect of family history has been analyzed by standard epidemiological methods. Such data are ideal to explore the validity of existing models for familial breast cancer. This paper explores the validity of the Jonker model that adds a hypothetical gene to the well-known BRCA1 and BRCA2 genes. The validity of the model for the Swedish data is checked by using a calibration model for breast cancer incidence given the (retrospective) family history as assessed at the end of the study period. This enables the validity of the overall incidence and the effect of family history to be assessed in the same model. The conclusion is that the existing model does reasonably well for the effect of family history but is seriously wrong for the early incidence rate. Therefore, the model is refitted in the Swedish data. Finally, the calibration of the refitted model is checked when using current family history as used in the epidemiological studies. The refitted Jonker model fits the data well and shows good agreement with the epidemiological findings.

Modelling the effect of baseline risk in meta-analysis: a review from the perspective of errors-in-variables regression.

In meta-analysis of clinical trials, investigating the relationship between the baseline risk and the treatment benefit is often of interest in order to explain the between trials heterogeneity with respect to treatment effect. The relationship is commonly described with a linear model taking into account the fact that the latent baseline risk is estimated from a finite sample and thus subjected to measurement error. Depending on the specific assumption about the latent baseline risks, two different classes of methods can be pursued. In the literature, it is commonly assumed that the latent baseline risks are sampled from a (normal) distribution. Such methods are often criticised for needing a distribution. Here, we propose the use of methods that require no distributional assumption on the baseline risks. A number of alternative methods are reviewed and are illustrated via simulation and by application to a published meta-analysis data.

Sensitivity analysis of state-transition models: how to deal with a large number of inputs.

State-transition models are employed to project future prevalence rates of risk factors and diseases within populations. Sensitivity analysis should be performed to assess the reliability of the results but often the number of inputs of the model is so huge, and running the model is so time-consuming, that not all methods of sensitivity analysis are practically available. Screening methods detect which inputs have a major influence on the outputs. We briefly review the available screening methods, and discuss one in particular, Morris' OAT Design. We applied the method under different assumptions to a module of the RIVM Chronic Diseases Model, where we projected the rates of never smokers, former smokers and current smokers in time up to the year 2050, based on smoking rates, start, stop and quit rates from 2003 and information on selective mortality in smokers from the literature. Different assumptions with regard to the interval of the inputs used for screeing led to different conclusions, especially with regard to the importance of quit and relapse rates versus initial prevalence rates. This should not to be read as a lack of validity of the method, but it shows that any sensitivity method cannot be automated in a form that runs without expert guidance on the ranges.

Denial and social and emotional outcomes in lung cancer patients: the protective effect of denial.

Denial is a well-known phenomenon in clinical oncology practice. Yet whether the impact of denial on patient well-being is beneficial or harmful remains unknown. The purpose of the current study is to investigate the relationship between denial and social and emotional outcomes in a large sample of lung cancer patients over an extended time period. Denial and social and emotional outcomes were measured in 195 newly diagnosed lung cancer patients. Four assessments were conducted over 8 months. The level of denial was measured using the Denial of Cancer Interview. Patient-reported social and emotional outcomes were measured using the EORTC-QLQ-30 and the HADS. Patients with a moderate or increasing level of denial over time reported better social outcomes (role functioning: p = 0.0036, social functioning: p = 0.027) and less anxiety (p = 0.0001) and depression (p = 0.0019) than patients with a low level of denial. The overall quality of life was better among lung cancer patients who displayed either moderate or increasing levels of denial compared with those who displayed low levels of denial (p < 0.0001). A certain level of denial in lung cancer patients can have a protective effect on social and emotional outcomes. Clinicians should take this into account when providing information about the illness and its prognosis.